Crowd-Sourced Text Annotation System for Use by Information Extraction Applications

ABSTRACT

The vocabulary of pertinent terms used to highlight/filter medical records in a text annotation system is continually updated based on user feedback. To maximize the effectiveness of this updating, feed-back is extracted from all users of the system, thereby providing a ‘group-sourced’ vocabulary of pertinent terms. As each user modifies the provided vocabulary of pertinent terms to customize the text annotation system to conform to the user&#39;s preferences, the modifications are collected and communicated to the provider of the vocabulary of pertinent terms. The provider of the vocabulary of pertinent terms assimilates the modifications implemented by each user of the word annotation system to determine whether to modify the vocabulary of pertinent terms for subsequent users of the common vocabulary of pertinent terms.

FIELD OF THE INVENTION

This invention relates to the field of medical information systems, andin particular to a system and method that uses user feedback to identifypertinent words or phrases for efficient and effective data extractionfrom text documents

BACKGROUND OF THE INVENTION

Effective medical diagnosis and treatment requires an assessment of apatient's current medical condition as well as the patient's medicalhistory. If the patient has been with the same primary practitioner foran extended period, the primary practitioner is likely to be aware ofthe patient's medical history; but as situations change, patients arelikely to need to visit either a new primary practitioner or, morecommonly, a practitioner from another discipline. These otherpractitioners do not have the benefit of the original practitioner'sexperience. In like manner, particularly with regard to medicalspecialties, the practitioner may see the patient infrequently, andcannot be expected to remember the medical history of each patient.

In such situations, the practitioner must spend time reviewing thepatient's medical history to assimilate the patient's current conditionbefore developing a diagnosis and prognosis. The task of reviewing apatient's medical history is facilitated by the availability ofelectronic medical records, and computer applications that facilitate anefficient review of these records. For example, a cardiologist mayrestrict the review of a patient's records to material that isidentified as being related to the cardiac system. Such systems,however, typically require that the medical record information beorganized in a particular manner, including, for example, specific HTML,fields that can be used to identify which records may be pertinent tothe practitioner's field.

Some of the patient's records may, however, be in free-form text, andmay include information that the practitioner may find to be pertinent.Having to read such text records, however, consumes the practitioner'stime, often without the disclosure of pertinent information.

Tools have been developed to facilitate an efficient review of free-formtext records, as illustrated in FIGS. 1A-1C. FIG. 1A illustrates adisplay of the free-form text record. FIG. 1B illustrates a display ofthis text provided by a text annotation system that highlights the wordsand phrases that are pertinent to the diagnosis of the patient(hereinafter ‘pertinent terms’), so that the practitioner's attention isdirected to these pertinent terms. FIG. 1C illustrates a display of onlythe pertinent terms in the free-form text to potentially further directthe practitioner's attention to these pertinent aspects in the patient'srecord.

In some embodiments, the display provides only the pertinent terms as inFIG. 1C, but when the practitioner “mouses over” the displayed text(i.e. moves the mouse pointer to within the display area of the text),the display changes to a display of the entire free-form text, such asillustrated in either FIG. 1A or 1B. Other methods of ‘selecting’ thedisplayed text for displaying some or all of the free-form text of thepatient record would be evident to one of skill in the art.

A key to the effectiveness of conventional text annotation systems isthe proper identification of such “pertinent terms”, and the optionsprovided to the practitioner for defining such pertinent terms. In someembodiments, the pertinent terms may be derived from a general medicalontology, or a specialized ontology for a particular medical specialty.These pertinent terms may be defined by the provider of the textannotation system, or developed by the provider based on interactionswith a medical facility or organization that is implementing such asystem. In some embodiments, the individual user of the system may amendor supplement the vocabulary of pertinent terms.

Even with an extensive vocabulary of pertinent terms, however, because afree-form text record is not necessarily constrained to conform to sucha defined vocabulary, and because the vernacular changes to thevocabulary may outpace the changes to the defined vocabulary, andbecause the creators of the vocabulary may not be actively engagedpractitioners, the identification of pertinent terms in a patient'srecord may omit some newly identified pertinent terms, or may be soinclusive as to minimize the effectiveness of the text annotation systemby highlighting minimally pertinent, or even irrelevant terms, therebyobscuring the actually pertinent terms.

SUMMARY OF THE INVENTION

It would be advantageous to provide a text annotation system thataccurately and reliably highlights pertinent terms in a patient'srecord. It would also be advantageous to provide a text annotationsystem that is able to keep pace with changing medical diagnostictechnology and vocabulary.

To better address one or more of these concerns, in an embodiment ofthis invention, the vocabulary of pertinent terms used tohighlight/filter medical records in a text annotation system iscontinually updated based on user feedback. To maximize theeffectiveness of this updating, feedback is extracted from all users ofthe system, thereby providing a ‘group-sourced’ vocabulary of pertinentterms. As each user modifies the provided vocabulary of pertinent termsto customize the text annotation system to conform to the user'spreferences, the modifications are collected and communicated to theprovider of the vocabulary of pertinent terms. The provider of thevocabulary of pertinent terms assimilates the modifications implementedby each user of the word annotation system to determine whether tomodify the vocabulary of pertinent terms for subsequent users of thecommon vocabulary of pertinent terms.

In one exemplary embodiment, a text annotation system is configured toreceive a vocabulary of pertinent terms from a provider that providesthe vocabulary to a plurality of practitioners, then processes a patientrecord to identify pertinent terms in the patient record based on thevocabulary of pertinent terms. The identified pertinent terms in thepatient record are displayed in a distinctive manner to thepractitioner, and the practitioner's modifications to the vocabulary ofpertinent terms are recorded. These modifications of the vocabulary arecommunicated to the provider of the vocabulary, along with modificationsfrom other users of this vocabulary. Thereafter, the text annotationsystem receives an updated vocabulary of pertinent terms from theprovider based on these modification of the vocabulary.

A crowd sourced knowledge module provides a common vocabulary ofpertinent terms to a plurality of text annotation systems, then receivesmodifications to the vocabulary of pertinent terms from the textannotation systems. The module assimilates the modifications to thevocabulary to determine whether an update to the vocabulary of pertinentterms is warranted, and if so, updates the vocabulary of pertinent termsand subsequently provides the updated vocabulary of pertinent terms tothe text annotation systems.

A network of text annotation systems is provided that comprises: adatabase that stores a vocabulary of pertinent terms that may be used ina medical record; a plurality of text annotation systems that eachhighlight pertinent terms in a patient's medical record based on thevocabulary of pertinent terms, and receives a user's proposedmodifications to the vocabulary of pertinent terms; and a crowd-sourcedknowledge module that provides the vocabulary of pertinent terms at thedatabase to the plurality of text annotation systems, receives theproposed modifications to the vocabulary of pertinent terms from theplurality of text annotation systems, assimilates the proposedmodifications to the vocabulary to determine whether an update to thevocabulary of pertinent terms is warranted, and updates the vocabularyof pertinent terms when the update is determined to be warranted.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is explained in further detail, and by way of example,with reference to the accompanying drawings wherein:

FIG. 1A illustrates an example free-form text medical record.

FIG. 1B illustrates an annotation of the free-form text of FIG. 1A tohighlight pertinent terms in the free-form text medical record.

FIG. 1C illustrates an alternative annotation of the free-form text ofFIG. 1A to display on the pertinent terms in the free-form text medicalrecord.

FIG. 2 illustrates an example network of text annotation systems thatshare a common vocabulary of pertinent terms that is continually updatedbased on feedback provided by the users of this network of textannotation systems in accordance with aspects of this disclosure.

FIG. 3 illustrates a flow diagram of an example use of a text annotationsystem that uses a vocabulary of pertinent terms that is commonly usedby a plurality of users, wherein the vocabulary of pertinent terms iscontinually updated based on feedback provided by the plurality of usersin accordance with aspects of this disclosure.

Throughout the drawings, the same reference numerals indicate similar orcorresponding features or functions. The drawings are included forillustrative purposes and are not intended to limit the scope of theinvention.

DETAILED DESCRIPTION

In the following description, for purposes of explanation rather thanlimitation, specific details are set forth such as the particulararchitecture, interfaces, techniques, etc., in order to provide athorough understanding of the concepts of the invention. However, itwill be apparent to those skilled in the art that the present inventionmay be practiced in other embodiments, which depart from these specificdetails. In like manner, the text of this description is directed to theexample embodiments as illustrated in the Figures, and is not intendedto limit the claimed invention beyond the limits expressly included inthe claims. For purposes of simplicity and clarity, detaileddescriptions of well-known devices, circuits, and methods are omitted soas not to obscure the description of the present invention withunnecessary detail.

This disclosure recognizes that reliance on a sole source of ‘knowledge’has substantial limitations. In the case of defining pertinent terms ina patient record, the definition of which terms are pertinent ornon-pertinent conventionally relies on the ‘knowledge’ of the person ororganization that creates the vocabulary of pertinent terms. Althoughsome systems may enable each user to customize the vocabulary inaccordance with that user's preferences, such customizations are limitedto that particular user's expertise in distinguishing between pertinentand non-pertinent terms, as contrast to the provided vocabulary ofpertinent terms.

For example, a novice practitioner may not recognize why a particularterm may be pertinent and may assume that the vocabulary is out-of-dateand modify his/her personal vocabulary to remove that term from thevocabulary of pertinent terms. In like manner, another practitioner mayhave had some experience that indicates to this practitioner that thevocabulary should be expanded to include a term that was omitted fromthe vocabulary of pertinent terms. In each case, the practitioner isconfident that, based on the practitioner's knowledge base, the removalfrom or addition to the vocabulary is warranted; but, from an objectiveviewpoint, whether or not these particular modifications are ‘valid’ isunknown.

Accordingly, if a system enables a user to modify the vocabulary ofpertinent terms, the user's modified vocabulary may not be consistentwith generally accepted definitions of pertinent and non-pertinentinformation. On the other hand, a system that is constrained to thevocabulary of pertinent terms that is defined by a provider of thevocabulary runs the risk of becoming outdated unless the provider of thevocabulary is vigilant in keeping the vocabulary up-to-date as theknowledge in the medical community grows or changes.

The inventor has recognized that a networking of practitioners havingaccess to a common vocabulary of pertinent terms enables the followingfeatures:

-   -   providing user flexibility in customizing the vocabulary;    -   enabling a validation of such customization; and,    -   providing a continuingly validated vocabulary of pertinent terms        to the plurality of users of this vocabulary in their text        annotations systems.

FIG. 2 illustrates such a network of practitioners having access to acommon vocabulary of pertinent terms. Each practitioner is provided atext annotation system 210 that includes a parser 220 that identifiespertinent terms in one or more patient's records 230. In an exampleembodiment, the patient's record(s) may include free-form text such asillustrated in FIG. 1A.

The parser 220 identifies each occurrence of a pertinent term in thepatient's record and displays the pertinent terms in a distinctivemanner, as illustrated in either of FIGS. 1B and 1C. If the initialdisplay is as illustrated in FIG. 1C, wherein only the pertinent termsare displayed, the practitioner may ‘select’ the display to enable adisplay of the full text, as illustrated in either of FIGS. 1A or 1B.

A vocabulary 260 of the pertinent terms may be commonly provided to allof the pertinence parsers 220 via the network 240. In accordance with anaspect of this disclosure, a crowd sourced knowledge module 250 isconfigured to update this vocabulary 260 based on feedback from theusers of the text annotation systems 210.

In an example embodiment, each user of the text annotation system 210 isprovided the option of locally modifying the vocabulary of pertinentterms, to identify, for example, terms in the art that have newly beenfound to be pertinent, or to remove terms that are no longer deemed tobe pertinent, and so on. When such modifications are made, themodifications are communicated to the crowd sourced knowledge module250. This communication may occur in real time, as the modifications aremade, or at periodic intervals, such as daily or at the end of eachwork-shift at the medical facility that provides the text annotationsystem 210.

The crowd sourced knowledge module 250 may be configured to assimilateall of the modifications received as they are received, or periodically,and determine what modifications, if any, should be made to the commonlyprovided vocabulary 260 based on the received modifications from theusers of the text annotation systems 210. In some embodiments, thevocabulary 260 is continuously updated; in other embodiments, thevocabulary 260 is updated based on the modifications received over adefined time period, such as every few hours, or daily.

After updating the vocabulary 260, the module 250 may ‘broadcast’ theupdated vocabulary 260 to each of the text annotation systems 210, orthe updated vocabulary 260 may be provided in response to specificrequests for the vocabulary 260 from each text annotation system 210.The broadcast may be scheduled to occur periodically, or whenever achange is made to the vocabulary 260.

Any number of techniques may be used to assimilate the modificationsreceived from the text annotation systems 210, typically based on atradeoff between the risk of omitting a pertinent term from thevocabulary 260 and the risk of obscuring the display of pertinent termsin a report with non-pertinent terms.

In some embodiments, a simple voting scheme may be used, wherein if moreusers choose to add a term to the pertinent vocabulary 260 than todelete the term from the pertinent vocabulary 260, the term is added tothe vocabulary 260; otherwise the term is deleted from the vocabulary260. If, on the other hand, the risk of omitting a relevant term fromthe vocabulary 260 is considered to be more significant that the risk ofpotentially including non pertinent terms in the vocabulary 260(hereinafter ‘cluttering’), the term may be added to the vocabulary 260whenever a user modifies the local vocabulary to include the term, butonly deletes a term from the vocabulary 260 when the number ofmodifications for removal of the term is significantly greater than thenumber of modifications for adding the term to the vocabulary 260.

In some embodiments, a weighted accumulation may be maintained wherein,for example, a modification to add a term to the vocabulary 260 may bevalued as a large positive number, and a modification to remove the termfrom the vocabulary may be valued as a small negative number, and theterm remains in the vocabulary 260 whenever the accumulation is greaterthan zero. If, on the other hand, the risk of cluttering is consideredto outweigh the risk of omission, the modifications to add a term may begiven a small positive value and the modifications to delete a term maybe given a large negative value.

In some embodiments, recent modifications may be considered to be moresignificant than older modifications, and a rolling average may bemaintained wherein more recent modifications are weighted more thanolder modification.

One of skill in the art will recognize that this crowd sourced feedbackscheme effectively provides a ‘sampling’ system for determining whichterms are likely to be considered pertinent by the general population ofusers of the text annotation systems 210. Accordingly, statisticaltechniques may also be used to assimilate the modification ‘samples’ soas to only provide changes to the vocabulary 260 when it can be shownthat the modifications are ‘statistically significant’.

The assimilation may also consider the effect of “non-modifications”.That is, for example, if a user does not modify the vocabulary whileusing the vocabulary 260 to review a patient's records, it may beassumed that the user agrees with the content of the vocabulary 260 withrespect to the identified pertinent terms in the review of the patient'srecords, and this ‘passive agreement’ should affect any decision todelete these terms from the vocabulary 260. In some embodiments, thefeedback provided from the text annotation system 210 may include a listof the pertinent terms that were displayed to the user (and not markedfor removal), or, more selectively, a list of the pertinent terms thatwere displayed in a window that the user selected to display the fulltext of the record. These terms may receive, for example, a smallpositive value in the aforementioned accumulation, thereby reducing thelikelihood of removing these ‘passively accepted’ terms until asubstantial number of users indicate a preference for their removal.Such a passive reinforcement of existing terms in the vocabulary 260 mayobviate the need to use the aforementioned weighted values to offset theeffect of modifications that would remove terms from the vocabulary 260.

The assimilation may also be configured to set a ‘threshold’ value forinitially adding a term to the vocabulary 260, to avoid unnecessary‘chatter’ when a single user modifies the vocabulary to include a newterm. Because this would be the only ‘vote’ regarding the term, in aconventional voting scheme this vote would likely result in a change tothe vocabulary 260 to include this term. However, this new term maylikely be considered non-pertinent by many of the other users, resultingin numerous subsequent modifications to remove the term from thevocabulary. To avoid such a situation, the assimilation may beconfigured to only add a new term to the vocabulary 260 when a givennumber of users modify the vocabulary to include this term. Thisminimum-number-of-users threshold scheme may also be effective inminimizing the risk of having a malicious user adding a “commonly used”term that will obviously lead to cluttering of the displays of allusers, at least until the feedback is received to remove this term.

One of skill in the art will recognize that this continuous feedback ofuser modifications to the crowd sourced knowledge module and subsequentupdating of the common vocabulary 260 of pertinent changes may producetransient effects as changes are made to the vocabulary 260, thencountermanded by reactionary feedback, but the eventual ‘stabilized’terms in the common vocabulary 260 are likely to be agreeable to themajority of users of the text annotation systems 210 that use thisvocabulary 260.

FIG. 3 illustrates a flow diagram of an example use of a text annotationsystem in accordance with aspects of this disclosure. The left columnindicates actions taken at the text annotation system, such as thesystem 210 of FIG. 2, and the right column indicates actions taken bythe crowd sourced knowledge module 250 of FIG. 2.

At 310, the module 250 provides the vocabulary of pertinent terms to thetext annotation system 210. This may be performed as a broadcast to allof the text annotation systems 210 in the network, or it may beperformed in response to a request for the vocabulary from the textannotation system 210.

At 320, the vocabulary of pertinent terms is downloaded at the textannotation system 210, and used, at 330, to parse a patient's record(s)and display any pertinent terms in the patient's record in a distinctivemanner, such as highlighted within a display of a free-text record (e.g.FIG. 1B), or displayed without the non-pertinent terms in the patient'srecord (e.g. FIG. 1C).

Optionally, at 340, the user may select the displayed information by,for example, placing a mouse pointer over the displayed information, orby ‘clicking’ or ‘double-clicking’ on the displayed information.

If the user desires to modify the vocabulary of pertinent terms, to addor delete a term, for example, the user may effect such a modificationat 350. In an example embodiment, the user may ‘right click’ on a termto change its status. If the term is currently a pertinent term in thevocabulary, its status is changed to non-pertinent and removed from thelocal copy of the vocabulary at the text annotation system; if the termis currently a non-pertinent term, its status is changed to pertinentand added to the local copy of the vocabulary. The text annotationsystem then updates the display of pertinent terms in the patient'srecord(s) based on this change to the local vocabulary.

At 360, any modifications to the local vocabulary are communicated fromthe text annotation system 210 to the crowd sourced knowledge module250. As noted above, the modifications may be communicated as they aremade, or at periodic or aperiodic intervals.

At 370, the modifications are received at the module 250, along with anymodifications from other text annotation systems 210. Thesemodifications are assimilated by the module 250, at 380, to determinewhether a change to the vocabulary of pertinent terms is warranted. Asdetailed above, a weighted or unweighted voting scheme, preferably withthresholding, may be used to determine the changes that are likely to beagreeable to the population of users of the text annotation systems 210.

At 390, the determined changes to the vocabulary of pertinent terms areimplemented, so that the next time the vocabulary is sent from themodule 250, at 310, it includes these changes.

As noted above, this disclosure provides an automated method of keepinga vocabulary of pertinent terms up-to-date, while at the same time,providing affirmation that each stabilized revision of the vocabulary islikely to be agreeable to a substantial majority of the users of thetext annotation systems that use this vocabulary.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive; theinvention is not limited to the disclosed embodiments.

For example, in an alternative embodiment, each user of the textannotation system 210 may maintain a ‘private’ supplemental vocabularythat augments the vocabulary 260 that is received from the crowd sourcedknowledge module 250. The supplemental vocabulary may be configured toidentify terms in the vocabulary 260 that are to be considerednon-pertinent, and to identify additional terms, which may or may not bein the vocabulary 260, that are always to be considered as pertinent.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfill thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage. A computer program may be stored/distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

1. A non-transitory computer readable medium that includes a programthat, when executed by a processing system, causes the processing systemto: receive a vocabulary of pertinent terms from a provider thatprovides the vocabulary to a plurality of practitioners; receive arequest from a practitioner for a patient record; process the patientrecord to identify pertinent terms in the patient record based on thevocabulary of pertinent terms; display the identified pertinent terms inthe patient record in a distinctive manner to the practitioner; receive,from the practitioner, a selection of a displayed pertinent term, and anidentification of a proposed modification to remove the selectedpertinent term from the vocabulary of pertinent terms; communicate theproposed modification of the vocabulary to the provider of thevocabulary; and subsequently receiving an updated vocabulary ofpertinent terms from the provider based on the proposed modification ofthe vocabulary by the practitioner and also based on one or moreproposed modifications from the plurality of practitioners to add a newterm to the vocabulary of pertinent terms, and one or more proposedmodifications to remove an existing term front the vocabulary ofpertinent terms.
 2. The medium of claim 1, wherein the program causesthe processor to: display one or more non-pertinent terms in thepatient's record, receive, from the practitioner, a selection of adisplayed non-pertinent term, and an other proposed modification thatthe selected non-pertinent term in the patient's record should be addedto in the vocabulary of pertinent terms, and communicate the otherproposed modification to the provider of the vocabulary,
 3. (canceled)4. The medium of claim 1, wherein the program causes the processor todisplay the identified pertinent terms in a distinctive manner bydisplaying only the pertinent terms.
 5. The medium of claim 4, whereinthe program causes the processor to subsequently display an entirecontent of at least a portion of the patient record when thepractitioner indicates selection of a display area containing thepertinent terms.
 6. The medium of claim 1, wherein the program causesthe processor to display the identified pertinent terms in a distinctivemanner by displaying the pertinent terms using a different displayformat than other terms in the patient record.
 7. A non-transitorycomputer readable medium that includes a program that, when executed bya processing system, causes the processing system to: provide avocabulary of pertinent terms to a plurality of text annotation systems;receive proposed modifications to the vocabulary of pertinent terms fromtwo or more of the plurality of text annotation systems, wherein theproposed modifications include modifications that add a term to thevocabulary and modifications that remove a term from the vocabulary;assimilate the proposed modifications to the vocabulary to determinewhether an update to the vocabulary of pertinent terms is warranted;updating the vocabulary of pertinent terms when the update is determinedto be warranted; and providing the updated vocabulary of pertinent termsto one or more of the plurality of text annotation systems.
 8. Themedium of claim 7, wherein the program causes the processor toassimilate the modifications by maintaining a count of the modificationsthat add the term to the vocabulary of pertinent terms and themodifications that remove the term from the vocabulary.
 9. The medium ofclaim 8, wherein the count of modifications is a weighted accumulation,wherein a first weight is applied to each modification that adds theterm to the vocabulary and a second weight is applied to eachmodification that removes the term from the vocabulary.
 10. The mediumof claim 8, wherein a non-zero threshold is applied to the count ofmodifications to determine whether an update to the vocabulary iswarranted.
 11. A network of text annotation systems comprising: adatabase that stores a vocabulary of pertinent terms that may be used ina medical record; a plurality of text annotation systems that each:highlight pertinent terms in a patient's medical record based on thevocabulary of pertinent terms; and receive proposed modifications to thevocabulary of pertinent terms by a user of the text annotation system; acrowd-sourced knowledge module that: provides the vocabulary ofpertinent terms at the database to the plurality of text annotationsystems; receives the proposed modifications to the vocabulary ofpertinent terms from the plurality of text annotation systems, whereinthe proposed modifications include modifications that add a term to thevocabulary and modifications that remove a term from the vocabulary;assimilates the proposed modifications to the vocabulary to determinewhether an update to the vocabulary of pertinent terms is warranted;updates the vocabulary of pertinent terms when the update is determinedto be warranted; and provides the updated vocabulary of pertinent termsto one or more of the plurality of text annotation systems.
 12. Thenetwork of claim 11, wherein at least one of the text annotation systemsdisplays one or more non-pertinent terms in the patient's record, andenables a user to select a non-pertinent term so as to indicate that theproposed modification is to add the select non-pertinent term to thevocabulary of pertinent terms.
 13. The network of claim 11, wherein atleast one of the text annotation systems enables a user to select adisplayed pertinent term so as to indicate that the proposedmodification is to remove the select pertinent term from the vocabularyof pertinent terms.
 14. The network of claim 13, wherein thecrowd-sourced knowledge module assimilates the proposed modifications bymaintaining a count of the modifications that add a term to thevocabulary of pertinent terms and the modifications that remove the termfrom the vocabulary.
 15. The network of claim 14, wherein the count ofmodifications is a weighted accumulation, wherein a first weight isapplied to each proposed modification that adds the term to thevocabulary and a second weight is applied to each proposed modificationthat removes the term from the vocabulary.
 16. The medium of claim 1,wherein the updated vocabulary of pertinent terms is based on a count ofthe modifications to add the selected pertinent term to the vocabularyof pertinent terms, and the modifications to remove the selectedpertinent term from the vocabulary,
 17. The medium of claim 16, whereinthe count of modifications is a weighted accumulation, wherein a firstweight is applied to each modification to add the selected remove theselected pertinent term from the vocabulary